Measuring Performance when Positives Are Rare: Relative Advantage versus Predictive Accuracy - A Biological Case Study

This paper presents a new method of measuring performance when positives are rare and investigates whether Chomsky like grammar representations are useful for learning accurate comprehensible predic tors of members of biological sequence families The positive only learn ing framework of the Inductive Logic Programming ILP system CPro gol is used to generate a grammar for recognising a class of proteins known as human neuropeptide precursors NPPs Performance is mea sured using both predictive accuracy and a new cost function Relative Advantage RA The RA results show that searching for NPPs by using our best NPP predictor as a lter is more than times more e cient than randomly selecting proteins for synthesis and testing them for biological activity Predictive accuracy is not a good measure of per formance for this domain because it does not discriminate well between NPP recognition models despite covering varying numbers of the rare positives all the models are awarded a similar high score by predictive accuracy because they all exclude most of the abundant negatives